Remove unnecessary CUDA sync of qwen image and video preprocess #22792
Conversation
![gemini-code-assist[bot]](https://avatars.githubusercontent.com/in/956858?s=60&v=4){kind=small}
There was a problem hiding this comment.
Code Review
This pull request aims to remove unnecessary CUDA synchronizations in qwen2_5_vl.py for performance improvement. The changes in _process_image_input and _process_video_input correctly identify a source of synchronization but introduce a correctness bug when handling empty inputs. My review provides critical feedback with suggestions to fix this bug while still achieving the desired performance gain.
|
👋 Hi! Thank you for contributing to the vLLM project. 💬 Join our developer Slack at https://slack.vllm.ai to discuss your PR in #pr-reviews, coordinate on features in #feat- channels, or join special interest groups in #sig- channels. Just a reminder: PRs would not trigger full CI run by default. Instead, it would only run Once the PR is approved and ready to go, your PR reviewer(s) can run CI to test the changes comprehensively before merging. To run CI, PR reviewers can either: Add 🚀 |
DarkLight1337
left a comment
DarkLight1337
left a comment
There was a problem hiding this comment.
Thanks, can you provide some profiling or benchmark results to show the effectiveness of this optimization?
|
@DarkLight1337 Added. |
Signed-off-by: cyy <cyyever@outlook.com>
Co-authored-by: Cyrus Leung <cyrus.tl.leung@gmail.com> Signed-off-by: Yuanyuan Chen <cyyever@outlook.com>
Signed-off-by: cyy <cyyever@outlook.com>
DarkLight1337
left a comment
DarkLight1337
left a comment
There was a problem hiding this comment.
Thanks for the quick response, LGTM!
github-actions
bot
added
the
ready
…-project#22792) Signed-off-by: cyy <cyyever@outlook.com> Signed-off-by: Yuanyuan Chen <cyyever@outlook.com> Co-authored-by: Cyrus Leung <cyrus.tl.leung@gmail.com> Signed-off-by: Diego-Castan <diego.castan@ibm.com>
…-project#22792) Signed-off-by: cyy <cyyever@outlook.com> Signed-off-by: Yuanyuan Chen <cyyever@outlook.com> Co-authored-by: Cyrus Leung <cyrus.tl.leung@gmail.com>
…-project#22792) Signed-off-by: cyy <cyyever@outlook.com> Signed-off-by: Yuanyuan Chen <cyyever@outlook.com> Co-authored-by: Cyrus Leung <cyrus.tl.leung@gmail.com>
…-project#22792) Signed-off-by: cyy <cyyever@outlook.com> Signed-off-by: Yuanyuan Chen <cyyever@outlook.com> Co-authored-by: Cyrus Leung <cyrus.tl.leung@gmail.com>
…-project#22792) Signed-off-by: cyy <cyyever@outlook.com> Signed-off-by: Yuanyuan Chen <cyyever@outlook.com> Co-authored-by: Cyrus Leung <cyrus.tl.leung@gmail.com> Signed-off-by: Xiao Yu <xiao.yu@amd.com>
…-project#22792) Signed-off-by: cyy <cyyever@outlook.com> Signed-off-by: Yuanyuan Chen <cyyever@outlook.com> Co-authored-by: Cyrus Leung <cyrus.tl.leung@gmail.com>
3 participants
Essential Elements of an Effective PR Description Checklist
This PR removes unnecessary CUDA sync in _process_image_input and _process_video_input of vllm/model_executor/models/qwen2_5_vl.py by utilising grid_thw_list.
Before the fix, the _process_image_input function of vllm/model_executor/models/qwen2_5_vl.py could take near 1/4 of the
OwnTimeas measured by py-spy, see:After the fix, the bottlenecks have been moved to other places, and there are further fixes. However, this fix is quite simple that I want to submit first.
